Variation in choice of study design: findings from the Epidemiology Design Decision Inventory and Evaluation (EDDIE) survey.

BACKGROUND: Researchers using observational data to understand drug effects must make a number of analytic design choices that suit the characteristics of the data and the subject of the study. Review of the published literature suggests that there is a lack of consistency even when addressing the same research question in the same database. OBJECTIVE: To characterize the degree of similarity or difference in the method and analysis choices made by observational database research experts when presented with research study scenarios. RESEARCH DESIGN: On-line survey using research scenarios on drug-effect studies to capture method selection and analysis choices that follow a dependency branching based on response to key questions. SUBJECTS: Voluntary participants experienced in epidemiological study design solicited for participation through registration on the Observational Medical Outcomes Partnership website, membership in particular professional organizations, or links in relevant newsletters. MEASURES: Description (proportion) of respondents selecting particular methods and making specific analysis choices based on individual drug-outcome scenario pairs. The number of questions/decisions differed based on stem questions of study design, time-at-risk, outcome definition, and comparator. RESULTS: There is little consistency across scenarios, by drug or by outcome of interest, in the decisions made for design and analyses in scenarios using large healthcare databases. The most consistent choice was the cohort study design but variability in the other critical decisions was common. CONCLUSIONS: There is great variation among epidemiologists in the design and analytical choices that they make when implementing analyses in observational healthcare databases. These findings confirm that it will be important to generate empiric evidence to inform these decisions and to promote a better understanding of the impact of standardization on research implementation.

Defining a reference set to support methodological research in drug safety.

BACKGROUND: Methodological research to evaluate the performance of methods requires a benchmark to serve as a referent comparison. In drug safety, the performance of analyses of spontaneous adverse event reporting databases and observational healthcare data, such as administrative claims and electronic health records, has been limited by the lack of such standards. OBJECTIVES: To establish a reference set of test cases that contain both positive and negative controls, which can serve the basis for methodological research in evaluating methods performance in identifying drug safety issues. RESEARCH DESIGN: Systematic literature review and natural language processing of structured product labeling was performed to identify evidence to support the classification of drugs as either positive controls or negative controls for four outcomes: acute liver injury, acute kidney injury, acute myocardial infarction, and upper gastrointestinal bleeding. RESULTS: Three-hundred and ninety-nine test cases comprised of 165 positive controls and 234 negative controls were identified across the four outcomes. The majority of positive controls for acute kidney injury and upper gastrointestinal bleeding were supported by randomized clinical trial evidence, while the majority of positive controls for acute liver injury and acute myocardial infarction were only supported based on published case reports. Literature estimates for the positive controls shows substantial variability that limits the ability to establish a reference set with known effect sizes. CONCLUSIONS: A reference set of test cases can be established to facilitate methodological research in drug safety. Creating a sufficient sample of drug-outcome pairs with binary classification of having no effect (negative controls) or having an increased effect (positive controls) is possible and can enable estimation of predictive accuracy through discrimination. Since the magnitude of the positive effects cannot be reliably obtained and the quality of evidence may vary across outcomes, assumptions are required to use the test cases in real data for purposes of measuring bias, mean squared error, or coverage probability.

Managing data quality for a drug safety surveillance system.

OBJECTIVE: The objective of this study is to present a data quality assurance program for disparate data sources loaded into a Common Data Model, highlight data quality issues identified and resolutions implemented. BACKGROUND: The Observational Medical Outcomes Partnership is conducting methodological research to develop a system to monitor drug safety. Standard processes and tools are needed to ensure continuous data quality across a network of disparate databases, and to ensure that procedures used to extract-transform-load (ETL) processes maintain data integrity. Currently, there is no consensus or standard approach to evaluate the quality of the source data, or ETL procedures. METHODS: We propose a framework for a comprehensive process to ensure data quality throughout the steps used to process and analyze the data. The approach used to manage data anomalies includes: (1) characterization of data sources; (2) detection of data anomalies; (3) determining the cause of data anomalies; and (4) remediation. FINDINGS: Data anomalies included incomplete raw dataset: no race or year of birth recorded. Implausible data: year of birth exceeding current year, observation period end date precedes start date, suspicious data frequencies and proportions outside normal range. Examples of errors found in the ETL process were zip codes incorrectly loaded, drug quantities rounded, drug exposure length incorrectly calculated, and condition length incorrectly programmed. CONCLUSIONS: Complete and reliable observational data are difficult to obtain, data quality assurance processes need to be continuous as data is regularly updated; consequently, processes to assess data quality should be ongoing and transparent.

Advancing the science for active surveillance: rationale and design for the Observational Medical Outcomes Partnership.

The U.S. Food and Drug Administration (FDA) Amendments Act of 2007 mandated that the FDA develop a system for using automated health care data to identify risks of marketed drugs and other medical products. The Observational Medical Outcomes Partnership is a public-private partnership among the FDA, academia, data owners, and the pharmaceutical industry that is responding to the need to advance the science of active medical product safety surveillance by using existing observational databases. The Observational Medical Outcomes Partnership's transparent, open innovation approach is designed to systematically and empirically study critical governance, data resource, and methodological issues and their interrelationships in establishing a viable national program of active drug safety surveillance by using observational data. This article describes the governance structure, data-access model, methods-testing approach, and technology development of this effort, as well as the work that has been initiated.

Mixed results in the safety performance of computerized physician order entry.

Computerized physician order entry is a required feature for hospitals seeking to demonstrate meaningful use of electronic medical record systems and qualify for federal financial incentives. A national sample of sixty-two hospitals voluntarily used a simulation tool designed to assess how well safety decision support worked when applied to medication orders in computerized order entry. The simulation detected only 53 percent of the medication orders that would have resulted in fatalities and 10-82 percent of the test orders that would have caused serious adverse drug events. It is important to ascertain whether actual implementations of computerized physician order entry are achieving goals such as improved patient safety.

Understanding hospital readiness for computerized physician order entry.

BACKGROUND: Many hospitals in the United States are in early stages of decision making and planning to implement computerized physician order entry (CPOE) to improve patient safety and quality of care. The targeted processes and the software for CPOE are complex, and implementation is a large-scale change effort for most hospitals. Hospitals can increase the likelihood of success by understanding and addressing gaps in CPOE readiness. ASSESSING CPOE READINESS: A CPOE readiness assessment tool was developed that includes several different components: external environment; organizational leadership, structure, and culture; care standardization;, order management; access to information; information technology composition; and infrastructure. The presence or absence of these indicators in a particular hospital was determined by on-site interviews, walkarounds with direct observations, and document review. RESULTS: Assessment results for the first 17 hospitals (bed size, 75-906 beds) indicated that the lowest average component score was in care standardization, while the highest average component score was in organizational structure and function. Organizational culture and the order management process also had low average scores. CONCLUSIONS: This CPOE readiness assessment revealed significant gaps in all the hospitals examined. Identifying these gaps and addressing them before CPOE implementation can reduce risks. Organizations need to develop expertise at accomplishing and sustaining change; understanding and building CPOE readiness is an important first step.